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The audio-video synchronous playback appa- 
ratus of the present invention tor expanding digital com- 
pressed audio data and digital compressed video data, 
respectively, and for synchronously playing back 
expanded audio data and expanded video data, com- 
prises: a means for obtaining an audio playk^ack time 
based on the amount of the played-lsack audio data; a 
means for determining progress in ®q>anding the video 

FIG.1 



data based on the audio playback time; and a recovery 
means for saving the processing time for the audio data 
arxi assigning saved time to a process for the video data 
so as to recover any delay in a video portion when the 
expansion process for the video data is delayed with 
respect to the expansion process for the audio data. 
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Description 

Field of the Invention 

[0001] The present invention relates to an audio-video synchronous playback apparatus for expanding digital com- 
pressed audio data and digital compressed video data, respectively, and for synchronously playing back the expanded 
audio data and the expanded video data. ^ ^. ^ 

[0002] This application is based on Japanese Patent Application No Hei 9-269423. the contents of which are incor- 
porated herein by reference. 

Background Art 

[00031 A conventional audio-video synchronous playback apparatus reads digital compressed audio data and digi^ 
compressed video data from a CD-ROM or a hard disc unit, expands them, and plays back the expanded audio data 
through a speaker and the expanded video data through a display device such as a CFIT monitor. ^. ^ 

[0004] Conventional audio-video synchronous playback apparatuses for expanding digital compressed audio data 
and digital compressed video data, respectively, and for synchronously playing back the expanded audio data and ttie 
expanded video data (hereinafter, this apparatus will be refen-ed to as the "playback apparatus") are divided broadly into 
two categories: the first category includes a playback apparatus which has a sufficient data throughput capacity or 
which handles small-sized compressed codes of video data, and the second category includes a playback appa^tj^ 
which has an insufficient data throughput capacity or which handles large-sized compressed codes of video data. With 
the playback apparatus in the first category, an audio data and video frames are played back without interrupton 
because the expansion processes for the audio data and for the video data are perfomied rapidly Meanwhile, with the 
playback apparatus in the second category, the video portion may be played back in slow motion and the audio is inter- 
mittently played back, not in real time manner, because the expansion process is perfomied slowly 
[0005] The audio portion which has been intenrtttently played back may often cause incompatibility to human senses, 
and the process for the audio data is therefore prior to the process for the video data so that the audio is continuous^ 
played back. The conventional apparatus assigns the portion of the throughput capacity, which has not been^gned 
to the expansion and playback processes for the audio data, to the expansion and playback processes for the video 
data When synchronizalion (coinciding of playback current positions) between the audio data and the video data is not 
established, a time lag may occur between the audio portion and the video portion, and an appropriate thinning process 
for the video frames (dropping of the video frames) has therefore been necessary. 

[0006] One known standaid for digital compression is MPEG (Moving Picture Experts Group), where multiplexed, 
compressed, and coded audio-video data are synchronously played back. In a compression process, time information 
for executing playback and display are attached to the audio data and ttie video data. In the expansion process, the 
playback is performed synchronizing ttie audto data witti the video data witti reference to ttie time information 
[0007] As is conventionally known, various types of synchronous playback methods for ttie audio data and ttie video 
data have been proposed. For example, one conventional mettiod isdisclosed in "An Image and Audio Decoder in Jap- 
anese Patent Application. First Publication No Hei 7-75059 (hereinafter, it is referred to as "Citation 1"). Wh«i audio 
playback precedes video playback on a time basis, ttie image and audio decoder interpolates the audio data in every 
block calculated based on a time delay, ttius synchronizing ttie audio data witti ttie video data. Further, wlien me video 
playback precedes ttie audio playback on a time basis, ttie image audto decoder continues displaying o( ttie video por- 
tion for the delay time, ttius synchronizing ttie audio data with ttie video data. o . 
[0008] Anottier method is disclosed in "An Animation Sound Synchronous Control Apparatus" in Japanese Patent 
Applicatton. First Publication Na Hei 7-1 10756 (hereinafter, it is referred to as "Citation 2"). The animation sound syn- 
chronous control apparatus sets ttie next images to be displayed based on a playback start time, a display proce» 
completion time, and the number of frames to be displayed per unit time (frame rale), so ttiat video playba* in accord- 
ance witti ttie frame rate can be provided. The animation sound synchronous control apparatus plays back data accom- 
panied by ttie video portion in audto playback, tiiereby synchronizing audto data witti ttie video data. 
[0009] However, ttie problem arises that, for example, when data having audto data which include background music 
and video data include important data, there is a possibility ttiat necessary frames may be dropped ttirough synchroni- 
zation of the audio data witii ttie video data by continuously dropping ttie video frames in ttie conventional meBiod. 
[001 0] According to Citation 1 , because ttie display or ttie playback is peribrmed in accordance with whichever is 
delayed, ttie playback may not be performed in a real-time manner, because the video portfon may be played back in 

rSllir' While in Citation 2 ttie frame rate is adjusted by skipping some video data during playback, some audto data 
are also skipped when the video data are skipped over because ttie audio data accompanies the video data. The audio 
portion may be intermittertly played back under conditions in which a CPU has low capacity. 
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SUMMARY OF THE INVENTION 

[001 2] It is therefore an object of the present invention to provide an audio-video synchronous playback apparatus 
which synchronizes audio data with video data even when the video playback is delayed with respect to the audio play- 
5 back. 

[0013] In order to accomplish the above object, an audio-video synchronous playback apparatus according to the 
present invention for expanding digital compressed audio data and digital compressed video data, respectively and for 
synchronously playing back expanded audio data and expanded video data, comprises: a means for obtaining an audio 
playback time based on the amount of the played-back audio data; a means for determining progress in expanding the 

10 video data based on the audio playback time; and a recovery means for saving the processing time for the audio data 
and assigning saved time to a process for the video data so as to recover any delay in a video portion when the expan- 
sion process for the video data is delayed with respect to the expansion process for the audio data. 
[001 4] In another aspect of the invention, the recovery means may comprise a means for degrading the tone quality 
of an audio portfon to save the processing time for the audio data. The recovery means may comprise a means for sWp- 

15 ping audio data to save the processing time for the audio data. The recovery means may comprise a means for moni- 
toring and limiting the number of the skips in a predetermined time when audfo data are skipped. The recovery means 
may comprise: a tone degrading means (first means) for degrading the tone quality of an audfo porton so as to save 
the processing time for the audfo data: an audio skipping means (second means) for skipping some audio data so as 
to save the processing time for the audio data: and a frame dropping means (third means) for dropping a video frame 

20 to recover any delay in the video portion when the delay is not recovered by the tone degrading means and said audio 

skipping means. x*u 
[001 51 According to the construction, even when a dock in the playback apparatus is not precise, the progress of the 
video portion can be determined by calculating a reference time for synchronization of the audio data wrth the video 
data based on the audio playback data amount calculating the ideal frame count based on the reference time, and 
25 comparing the ideal frame count with the actual frame count. When the video portion is delayed with respect to the 
audfo portion, the process for the audio data is saved and more time is assigned to the process for the video data, so 
thai the delay in the video portion with respect to the audio portion can be recovered and synchronization of tiie audio 
data with the video data can be achieved. 

[001 61 In playback of data in which tiie audio portion includes background music and the video porton includes impor- 
30 tant data, the processes for the audio data are omitted to recover tiie delay, thereby decreasing tiie possibility of drop- 
ping necessary frames through the synchronization of the audio data with the video data. 

BRIEF DESCRIPTION OF THE DRAWINGS 

35 [00171 

Fig. 1 is a block diagram showing a construction of an audio-video synchronous playback apparatus according to 
an embodiment of tiie present inverttion. 
Fig. 2 is a diagram showing a construction in MPEG audio data. 
40 Fig. 3 Is a flow chart showing a playback process in the audio-video synchronous playback apparatus of Rg. 1 . 
Figs. 4A and 4B are flow charts showing the audio process in Fig. 3 in more detail, 
ng. 5 IS a flow chart showing the audio process in Fig. 3 in more detail. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

45 

[001 8] Data construction handled in tiie present Invention will be explained by way of MPEG audfo data as an exam- 
ple. As shown in Fig. 2. MPEG audio data comprise AAUs (Audio Access Units), whfoh are tiie minimum units expand- 
able into audio signals independentiy An AAU comprises 32 bits of header information. 16 bits of error checks (options), 
audio data, and ancillary data. The top 12 bits in tiie header information, which are called a synchronous word, are set 
so to "1 and used for detection of the AAU. The header information further includes data formats such as the number of 
channels and a sampling frequency. 

[0019] Refen-ing to Fig. 1. the audio-video synchronous playback apparatus, according to an embodiment off tiie 
present invention, will be explained. 

[00201 The audio-video synchronous playback apparatus for synchronously playing back audio and video signals read 
55 from a data recorder 10. comprises an audio-video (AV) separator 20. a video processor 30. an audfo processor 40. an 
AV synchronization controller 50, a CRT monitor 60. and a speaker 70. 

[00211 The data recorder 10 stores tiie digital corrpressed coded audio-video data, in which digital compressed 
coded audio data and digital compressed coded video data are multiplexed and recorded as compressed audio-video 
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data. The AV separator 20 reads the compressed audio-vkdeo data from the data recorder 1 0. and separates audio data 
and video data from the multiplexed conpressed audio-video data. The video processor 30 expands the video data 
separated by the AV separator 20. The audio processor 40 expands the audio data separated by the AV separator 20. 
The AV synchronization controlter 50 performs syrwhronization control for the audio data and the video data. The CRT 
monitor 60 displays the video data expanded by the video processor 30. while the speaker 70 plays back the audio data 
expanded by the audio processor 40. 

10022] The video processor 30 coiTV)rises a compressed video buffer 31 . a video decoder 32. a frame buffer 33. and 
video display controller 34. The compressed video buffer 31 stores the video data, which are separated by the AV sep- 
arator 20 as compressed video data. The video decoder 32 analyzes attribute information (video header information) 
in the compressed video data stored in the compressed video buffer 31 . expands the compressed video data, and out- 
puts expanded video data. The frame buffer 33 stores the video data expanded by the video decoder 32. The video dis- 
play controller 34 sends the expanded video data from the frame buffer 33 to the CRT monitor 60. 
[0023] The audio processor 40 comprises a compressed audio buffer 41 . an audio decoder 42. a PCM buffer 43. and 
an audio playback controller 44. The compressed audio buffer 41 stores the audio data, which are separated by the AV 
separator 20. as compressed audio data. The audio decoder 42 analyzes attribute information (audio header informa- 
tion) in the compressed audio data stored in the compressed audio buffer 41 . expands the compressed audio data, and 
outputs expanded audio data. The PCM buffer 43 stores the audio data expanded by the audio decoder 42. The audio 
playback controller 44 sends the expanded audio data in the PCM buffer 43 to the speaker 70. 
[0024] The AV synchronization controller 50 comprises an audio data counter 51. a video frame counter 52. a clock 
generator 53. a delay detector 54. an audio tone controller 55. an audio sWp controUer 56, an audio skip block 57. and 

a frame drop controller 58. , ^ i.. i, ^ 

[0025] The audio data counter 51 stores an accumulated amount of audio data which have been played back and 
expanded from the beginning of the playback up to the present. The ckjck generator 53 calculates an audio playback 
elapsed time (reference time) based on the accumulated data amount stored in the audio data counter 51 and on the 
audio header information obtained in the expansion by the audio decoder 42. That is. the audio data counter 51 and ttie 
dock generator 53 work as a means for obtaining the playback time of the audio portion based on the playback audio 

data amount. . ^ * ^.-^ 

[0026] The video frame counter 52 stores the accumulated frame count (actual frame count) of the video data which 
have been expanded and displayed from the beginning of the playback up to the present. The delay detector 54 calcu- 
lates a video frame count to be naturally expanded and displayed (ideal frame count) based on the reference time 
obtained by the clock generator 53 and the video header information obtained by the video decoder 32. and compares 
the ideal frame count with the actual frame count from *ie video frame counter 52 to determine progress of video. TTial 
is, the video frame counter 52 and the delay detector 54 work as a means for determining the progress of the expansion 
process for the video data based on the audio playtjack time. , . . , ^ 

[0027] The audio tone controller 55 performs one of the following operations when the video playback is delayed with 
respect to the audio playback, and when the video playback precedes the audio playback or is synchronized with the 
audio playback. In the former case, the audio tone controller 55 directs the audio decoder 42 to tower a tone quality by 
reducing the number of audio channels when the audio data include a plurality of diannels. or by reducing a sampling 
frequency when the sampling frequency is a high frequency That is. the audio tone controller 55 vrorks with the audio 
decoder 42 as a meais for saving processing time for the audio data by degrading the audio quality (tone quality) when 
the expansion process for the video data is delayed with respect to the expansion process for the audio data, and to 
recover the delay of the video portion by assigning the saved time to the process for the video data In the latter case, 
the audio tone controller 55 directs the audio decoder 42 to perfonn the expansion process according to the audio 
header infornnation when the tone quality is degraded. _ 
[0028] The audio skip block 57 skips the expansion process for the audio data for one AAU in response to a skip com- 
mand which will be descrtoed below, and stores the audio data in which the last AAU is ©<panded. the data being 
stored in the PCM buffer 43, into the PCM buffer 43. That is. when the expansion process for the video data is delayed 
with respect to the expansion process for the audio data, the audio skip block 57 works with the audio decoder 42 mti 
the PCM buffer 43 as a means for recovering the delay in the video portion by skipping some audio data to save the 
processing time for the audo data and by assigning the saved time to the process for the video data. 
[0029] m response to a skip determination command from the delay detector 54, the audto skip controller 56 sends 
the skip command to the audio skip block 57. counts the nuntoer of the executed skips for the audio data in the audio 
skip block 57. and limits the number of skips executable in a specified time. In other words, the audio sk^p controller 56 
works as a means for monitoring and limiting the number of skips executable in a specified time. 
[0030] In response to a frame drop command from the audto skip controller 56. the frame drop controller 58 controls 
dropping of frames of the video data. That is. the frame drop controUer 58 works with the video decoder 32 as a means 
for recovering the delay in the video portion by dropping video frames when the expansion process for the video data 
is deljyed with respect to the expansion process tor the audto data, and when the delay in the video portion is not recov- 



4 



: <EP P0O7177A2J_> 



EP0 907 177 A2 



10 



ered by the skip for the audio data. 

[0031] Referring to Fig. 3. playback operation with the audio-video synchronous playback apparatus of Rg. 1 will be 
explained. 

[00321 In step A1 . multiplexed compressed audio-video data are read from the data recorder 1 0, the AV separator 20 
separates audio data and video data, the video data are stored in the compressed video buffer 31 . and the audio data 
are stored in the compressed audio buffer 41 . In st^ A2. the AV synchronization controller 50 and the audio processor 
40 perform the synchronization control process, the expansion process for the audio data, and the playback process. 
In step A3, the AV synchronization controller 50 and the video processor 30 perform the synchronization control proc- 
ess and the expansion/display process for the video data. In step A4. it Is delennined whether unprocessed com- 
pressed audio-video data remain in the data recorder 10. and the flow returns to step A1 when data remains. When no 
data remain, the playt)ack process for the compressed audio-video data is stopped. 
[0033] Refen-ing to Figs. 4A and 4B. the process in step A2 shown in Fig. 3 will be explained in detail. In step B1 . the 
audio data counter 51 obtains the accumulated amount of the audio data which have been played back. The flow pro- 
ceeds from step B1 to step 82. where the dock generator 53 calculates the playback elapsed time (reference time) from 
15 the beginning of the audio playback up to the present based on the audio header information obtained in step 88. which 
is described below, and on the accumulated amount of the audio playback data obtained in step 81 . The flow proceeds 

from step 82 to B3. ^ ^ ^. , ^ 

[0034] In step 83. the delay detector 54 calculates the video frame count to be naturally expanded and displayed 
(ideal frame count), based on the reference time obtained in step 82 and on the video header information obtained in 
20 Step C5, which is described below. The delay detector 54 obtains the frame count actually expanded and displayed 
(actual frame count), and compares the ideal frame count with the actual frame count. The flow proceeds from step 83 
to step 84. where rt determines the progress of the video playback with respect to the audio playback according to the 
result of comparison of the ideal frame count and the actual frame count When the video playback precedes the audio 
playback on a time basis or when they are synchronized, the flow proceeds to step 85. When the audio playback is 
25 delayed from the video playt^acK the flow proceeds to step 81 1 

[0035] In step 85. it is determined whether the tone quality has been degraded. When the tone quality has been 
degraded, the flow proceeds to step 86. When the tone quality has not been degraded, the flow proceeds to step 88. 
[0036] In step 86. the audio tone controller 55 upgrades the tone quality. The "upgrade of the tone quality" means 
directing the audio decoder 42 to perform the expansion process with the number of output channels or with a sampling 
30 frequency according to the audio header information. The flow proceeds from step 86 to step 87. 

[0037] In step 87. the dock generator 53 stores the present reference time as a former reference time and resets the 
audio data counter 51 . because the result of calculation for the reference time is changed according to the change of 
the tone quality by the audio tone controller 55. The next reference time is obtained by adding the former reference time 
as an offset value to a newly calculated reference time. The flow proceeds from step 87 to step 88. 
35 [0038] In step 88. the compressed audio data are read from the compressed audio buffer 41 . and are expanded by 
the audio decoder 42. Further, the audio decoder 42 carries out an analysis of the audio header information. The flow 
proceeds from step 88 to step 89. where the audio data expanded by the audio decoder 42 are stored into the PCM 
buffer 43. The flow proceeds from step 89 to 81 0. where the audio playback controller 44 controls the playback for the 
expanded audio data stored in the PCM buffer 43. 
40 [0039] In step 81 1 . it is determined whether the tone quality has been degraded. When the tone quality has not been 
degraded, the flow proceeds to step 81 2, and when the tone quality has been degraded, the flow proceeds to step 813. 
[0040] In step 812. the audio tone controller degrades the tone quality. The "degrading of the tone quality" means 
directing the audio decoder 42 to output the audio portion through a single channel when a plurality of the output chan- 
nels depend on the audio header information. It may be possible to direct the audio decoder 42 to reduce a sampling 
45 frequency by thinning out the audio data on a time basts when the sampling frequency is a high frequency. The flow 
proceeds from step 812 to step 87. 

[0041 1 In step 81 3. the audio skip controller 56 obtains the number of skips for the audio which are earned out in a 
predetermined time. The flow proceeds from step 813 to step 814. where the audio skip controller 56 compares the 
number of the executed skipped processes for the audio data with a preset maximum value of the number of skipped 
50 processes executable in a predetermined time. When the number of the skipped processes is greater than the maxi- 
mum value, the flow Proceeds to step 81 5. When the number of the skipped processes is less than the maximum value, 
the flow proceeds to step 88. 

[0042] In step 15. a flag of an instruction to drop the video frames is set because the audio portion becomes 
extremely deteriorated when the process for the aixJio data is further omitted. The flow proceeds from step 815 to step 

55 88. 

[0043] In step 81 6. the audio skip block 57 skips the expansion process for the audio data by one AAU. and the audio 
data in which the former AAU is expanded, the data being stored in the PCM buffer 43. are stored in the PCM buffer 43. 
The flow proceeds from step 816 to step 810. 
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[0044] Referring to Fig. 5. the process in step A3 of Rg. 3 will be explained in detail. In step C1 , the audio counter 51 
obtains the accumulated amount of the audio data which have been played back The flow proceeds from step CI to 
step C2, where the dock generator 53 calculates an audio playback elapsed time (reference time) from the beginning 
of the playback up to the present, based on the audio header Infbrmatwn obtained in step B8 described above and on 
the accumulated data amount of the played-back audio portion. The flow proceeds from step C2 to step C3. 
[0045] In step C3, the delay detector 54 calculates a video frame count to be naturally expanded and displayed (ideal 
frame count), based on the reference time obtained in step C2 and on the video header information obtained in step 
C5. whidi will be described below. Further, the delay detector 54 obtains the frame count actually expanded and dis- 
played (actual frame count), and compares the ideal frame court with the actual frame count. The flow proceeds from 
step C3 to step C4. 

[0046] In step C4, the progress of the video playback with respect to the audio playback is determined according to 
the result off comparison of the ideal frame count and the actual frame count in step C3. When the video playt)ack pre- 
cedes the audio playback on a time basis, the video processing is stopped. When the video playback is synchronized 
with the audio playback, the flow proceeds to step C5. When the video playback is delayed with respect to the audio 
playback the flow proceeds to step C8. 

[0047] In step C5, the compressed video data are read from the compressed video buffer 31 , and the video decoder 
32 expands the compressed video data. Further, the video decoder 32 analyzes the video header information. The flow 
proceeds from step C5 to step C6, where the video data expanded by the video decoder 32 are stored in the frame 
buffer 33. The flow proceeds from step C6 to step C7, where the video display controller 34 executes a display process 
for the expanded video data in the frame buffer 33. 

[0048] In step C8. it is determined whether the flag of the instruction to drop the video frame is set in step B1 5. When 
the flag is set, the f tow proceeds to step C9. When the flag is not set. the flow proceeds to step C5. 
[0049] In step C9, the frame drop controller 58 executes the frame drop process, which means directing the video 
decoder 32 to skip the object video frame. 

[0050] A method for determining synchronization when the MPEG data are played back with the audio-video synchro- 
nization playt^ck apparatus according to the present invention will now be explained. 

[0051] On the assumption that a layer in the audio header information of the compressed data is "layer r. the bit rate 
is 192,000 bit/second, the sampling frequency is 44.100Hz. and there are two channels for stereo sound, a format of 
the PCM data produced by expanding the data includes two channel, a sampling frequency of 44.100 Hz, and 16 
bits/sample. The clock generator 53 calculates the reference time based on the accumulated data amount obtained by 
the audio data counter 51 according to the formula (1): 

reference time = accumulated data amount / (sampling 
frequency x bit / 8[bits] x number of channel) = accumulated data amount / (44100 x 16 / 8 x 2). 

[0052] Because the result of calculation for the reference time is affected by the change off the tone quality, the present 
reference time is stored as the former reference time, and the accumulated data amount is reset. The clock generator 
53 calculates the new reference time after the change of the tone quality according to the formula (2): 

reference time = accumulated data amount / (sampling 
frequency x bit / 8[bits] x number of channel) + former reference time. 

[0053] When the bit rate in the video header information is 1 .400.000 bits/second and the frame rate is 30 frames/sec- 
ond, the delay detector 54 calculates the ideal frame count based on the reference time according to the formula (3): 

ideal frame count o reference time x frame rale » reference time x 30. 

[0054] Next, the delay detector 54 compares the actual frame count obtained by the video frame counter 52 with the 
kieai frame count The progress of the video portion with respect to the audio portion is calculated according to the for- 
mula (4): 

progress of video portion = actual frame count - ideal frame count. 

[0055] The determination of progress of the vkleo portion will be explained tsy substituting numerical values for the 
ideal frame count and the actual frame count. 

[0056] When the kieal frame count is 10 and the actual frame court is 1 1 . the progress of the video portion is deter- 
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mined as 1 and it is determined that the video portion is fester than the audio portion. In this case, the video process is 
not performed. 

[00571 When the ideal frame count is 1 0 and the actual frame count is 1 1 , the progress of the video portion is deter- 
mined as 0 and it is determined that synchronization has been achieved. The process proceeds to step C5, where the 

5 expansion and display processes for the video data are carried out. 

[00581 When the ideal frame count is 10 and the actual frame count is 5. the progress of the video portion is deter- 
mined as -5, and it is determined that the video portion is delayed with respect to the audio portion. The delay is recov- 
ered by degrading the tone quality in the audio process and skipping the audio portion so as to save time for the process 
for the audio data, and by assigning the saved time to the process for the video data. In the video process, when the 

10 flag of the instruction to drop the video frame is set in step B1 5, the delay in the video portion is recovered by dropping 
the video frames. 

[00591 The degrading of the tone quality will be e)q3lained. In the invention, the degrading is achieved by playing back 
multichannel data through a single channel. To convert two channels of data, which are a right channel and a left chan- 
nel, into one channel of data, average values of data in the right and left channels are used. The data obtained through 

15 a single channel are half the data size of the original data through two channels. 

[00601 Further, in the inventton. the degrading of the tone quality is achieved by reducing the sampling frequency For 
example, when the data with the sampling frequency of 44,100 Hz are converted into data with a sampling frequency 
of 22.050 Hz. the conversion is achieved by executing a thinning process in every one sample. The data obtained at 
22,050 Hz are haH the data size of the original data at 44,100 Hz. 

20 [0061 1 The above described two methods may be combined to reduce the data size to 1 /4, The time required for data 
transmission between modules and devices can be reduced to 1/4, compared with the fonner state, and the saved time 
can be assigned to the process for the video data. 

[00621 The audio skipped process wBI be explained. The playback time for one AAU in the MPEG data is calculated 
according to the formula (5): 

25 

time for one AAU = number of sample x bit rate / 
(sampling frequency x bit rate) x 1000 - 384 x 192000 / (44100 x 192000) x 1000 =8.707 milliseconds 

30 [00631 The invention utilizes tiie auditory characteristic that people cannot sense a skip of one AAU in tiie invention 
by hearing because the time for one AAU is extremely short. However, when too many audio skipped processes are 
caried out in a predetermined time, the sound may become slow and flat. To avoid this, the maximum number of exe- 
cutable skips in a predetermined time is preset, and when the number of skips reaches the maximum, tiie flag of ttie 
instruction to drop video frames is set. 

35 [00641 As described above, ttie audio skipped process saves tiie time for expansion of tiie audio data, and tiie saved 
time can be allocated to tiie process for tiie video process. 

[00651 VVhile ttie preferred embodiment of tiiis invention has been described above, obviously nrxxliffications and var- 
iations are possible within ttie scope of this invention. For example, while tiie embodiment uses the reduction of the 
number of the channels and of the sanpling frequency as tiie mettiod for degrading tiie tone quality, otiier mettiods for 

40 reducing an output data size may be employed. While the embodiment uses the average values in ttie metiiod for 
reducing the number of ttie channels, other reduction methods may be employed. Further, while the embodiment uses 
the ttiinning process in ttie mettiod for reducing the sampling frequency, otiier reduction mettiods may be employed. 
[00661 This invention may be embocfied in ottier forms or carried out in otiier ways witiiout departing from tiie spirit 
thereof. The present embodiments are therefore to be considered in all respects illusti-ative and not limiting, the scope 

45 of ttie invention being indicated by tiie appended claims, and aD modifications falling wittiin the meaning and range of 
equivalency are intended to be embraced therein. 

Claims 

so 1. An audio-video synchronous playback apparatus for expanding digital compressed audio data and digital com- 
pressed video data, respectively, and for synchronously playing back expanded audio data and expanded video 
data, comprising: 

means for obtaining an audio playback time based on the amount of said played-back audio data; 
55 means for determining progress in expanding said video data based on said audio playljack time: and 

recovery means for saving ttie processing time for said audio data and assigning saved time to a process for 
said video data so as to recover any delay in a video portion when ttie expansion process for said video data 
is delayed witii respect to the expansion process for said audio data 
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2. An audio-video synchronous playback apparatus according to claim 1, wherein said recovery means comprises 
means for degrading the tone quality of an audio portion so as to save said processing time for said audio data. 

3. An audio-video synchronous playback apparatus according to claim 1, wherein said recovery means comprises 
means for skipping said audio data so as to save said processing time for said audio data. 

4. An audio-video synchronous playback apparatus according to claim 3. wherein said recovery means comprises 
means for monitoring and limiting the number of the skips in a predetermined time when said audio data are 
skipped. 

5. An audio-video synchronous playback apparatus according to claim 1 , wherein said recovery means comprises: 

tone degrading means for degrading the tone quality of an audio portion so as to save said processing time for 
said audio data; 

audio skipping means for skipping said audio data so as to save said processing time for said audio data; and 
video dropping means for dropping a video frame to recover any delay in the vkJeo portion when the delay is 
not recovered by sakJ tone degrading means and said audio skipping means. 

6. An audio-video synchronous playback method for expanding digital compressed audio data and digital com- 
pressed video data, respectively, and for synchronously playing back expanded audio data and expanded video 
data, comprising the steps of: 

obtaining an audio playback time based on the amount of said played-back audio data; 

determining progress in expanding said video data based on said audio playback time; and 

saving processing time for said audio data and assigning saved time to a process for sakj video data so as to 

recover any delay in a video portion when the expansion process for said video data is delayed with respect to 

the expansion process for said audio data. 

7. An audio-video synchronous playback method according to claim 6, wherein the step of recovering any delay in a 
video portion indudes degrading the tone qualrty of an audio portion so as to save said processing time for said 
audio data. 

8- An audio-video synchronous playback method according to claim 6, wherein the step of recovering any delay in a 
video portion includes skipping said audio data so as to save said processing time for said audio data. 

9. An audio-video synchronous playfc>ack method according to claim 8, wherein the step of recovering delay in a video 
portion includes monitoring and limiting the number of the skips in a predetermined time when said audio data are 
skipped. 

10- An audio-video synchronous playt>ack method according to claim 6. wherein the step of recovering any delay in a 
video portion includes the steps of: 

degrading the tone quality of an audio portion so as to save said processing time for said audio data; 
skipping said audio data so as to save said processing time for said audio data; and 

dropping a video frame to recover any delay in the video portion when the delay is not recovered by said first 
and second means. 

1 1 . An audio-video synchronous playback metiiod according to claim 7, wherein the tone quality of tiie audio portion is 
degraded by reducing at least one of the number of audio channels, and the sampling frequency. 

12. An audio-video synchronous playt^ack metiiod according to claim 6. wherein tiie progress in expanding said video 
data is determined by calculating an ideal frame count of video frames to be naturally expanded and displayed, and 
by comparing an actual frame count of video frames which has been actually expanded and displayed, witii said 
ideal frame courrt. 
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